Instead, we apply powerful machine learning models for one-class (Moya et al., 1993) or

Neural Information Processing Systems http://nips.cc/

In particular, they tend to be overconfident.

Ensuring factuality is essential for the safe use of Large Language Models (LLMs) in high-stakes domains such as medicine and law. Conformal inference provides distribution-free guarantees, but existing approaches are either overly conservative, discarding many true-claims, or rely on adaptive error rates and simple linear models that fail to capture complex group structures. To address these challenges, we reformulate conformal inference in a multiplicative filtering setting, modeling factuality as a product of claim-level scores. Our method, Multi-LLM Adaptive Conformal Inference (MACI), leverages ensembles to produce more accurate factuality-scores, which in our experiments led to higher retention, while validity is preserved through group-conditional calibration. Experiments show that MACI consistently achieves user-specified coverage with substantially higher retention and lower time cost than baselines. Our repository is available at https://github.com/MLAI-Yonsei/MACI

Conformal inference provides a general distribution-free method to rigorously calibrate the output of any machine learning algorithm for novelty detection. While this approach has many strengths, it has the limitation of being randomized, in the sense that it may lead to different results when analyzing twice the same data and this can hinder the interpretation of any findings. We propose to make conformal inferences more stable by leveraging suitable conformal e-values instead of p-values to quantify statistical significance. This solution allows the evidence gathered from multiple analyses of the same data to be aggregated effectively while provably controlling the false discovery rate. Further, we show that the proposed method can reduce randomness without much loss of power compared to standard conformal inference, partly thanks to an innovative way of weighting conformal e-values based on additional side information carefully extracted from the same data. Simulations with synthetic and real data confirm this solution can be effective at eliminating random noise in the inferences obtained with state-of-the-art alternative techniques, sometimes also leading to higher power.

Conformal prediction has emerged as a powerful framework for constructing distribution-free prediction sets with guaranteed coverage assuming only the exchangeability assumption. However, this assumption is often violated in online environments where data distributions evolve over time. Several recent approaches have been proposed to address this limitation, but, typically, they slowly adapt to distribution shifts because they update predictions only in a forward manner, that is, they generate a prediction for a newly observed data point while previously computed predictions are not updated. In this paper, we propose a novel online conformal inference method with retrospective adjustment, which is designed to achieve faster adaptation to distributional shifts. Our method leverages regression approaches with efficient leave-one-out update formulas to retroactively adjust past predictions when new data arrive, thereby aligning the entire set of predictions with the most recent data distribution. Through extensive numerical studies performed on both synthetic and real-world data sets, we show that the proposed approach achieves faster coverage recalibration and improved statistical efficiency compared to existing online conformal prediction methods.

Conformal inference [Vovk et al., 2005] provides a general procedure for constructing prediction sets with guaranteed coverage, under the assumption that the data are exchangeable. This assumption, however, is often too restrictive: it typically fails in sequential or time-dependent settings such as time series forecasting, where the distribution of observations may shift over time. To address this issue, Gibbs and Cand` es [2021] introduced Adaptive Conformal Inference (ACI), which extends Conformal Prediction (CP) to adversarial environments. ACI adapts to distribution shifts by updating prediction intervals in response to observed outcomes, ensuring that the empirical coverage converges to the desired level. While effective in maintaining coverage, ACI and its extensions generally lack efficiency guarantees-for instance, there is no control over the average length of prediction intervals in adversarial regimes. In this work, we study sequential conformal inference as a repeated two-player finite game and invoke Blackwell's theory of approachability to characterize feasible objectives. Building on this perspective, we design a calibration-based algorithm that ensures asymptotic validity while achieving asymptotic efficiency under mild assumptions. Our approach relies on the notion of opportunistic approachability [Bernstein et al., 2014], which allows the learner to exploit potential restrictions in the opponent's play. We argue that such assumptions better fit the typical use cases of ACI-such as distributional drift or regime switching-than the fully adversarial setting.

Neural Information Processing Systems http://nips.cc/